{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第五周作业\n",
    "数据：Million Song Dataset(MSD)    \n",
    "https://labrosa.ee.columbia.edu/millionsong/    \n",
    "\n",
    "音乐数据集 Million Song Dataset(MSD) ， 包含来自 SecondHandSongs dataset 、 musiXmatch dataset、Last.fm dataset、Taste Profile subset、thisismyjam-to-MSD mapping、tagtraum genre annotations 和 Top MAGD dataset 七个知名音乐社区的数据。   \n",
    "\n",
    "原始数据集包括： \n",
    "1. train_triplets.txt：三元组数据（用户、歌曲、播放次数） \n",
    "2. track_metadata.db：每个歌曲的元数据    \n",
    "由于原始数据太大，作业用的数据集只是其中的子集（播放次数最多的10万个用户、播放次数最多的3万首歌曲。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 导入必要的工具包\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from collections import defaultdict\n",
    "import scipy.io as sio\n",
    "import itertools\n",
    "import _pickle as cPickle\n",
    "# svd用到的包\n",
    "from numpy import mat, nonzero, shape, eye  \n",
    "from numpy import linalg as la   # Linear Algebra包"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>song</th>\n",
       "      <th>listen_count</th>\n",
       "      <th>title</th>\n",
       "      <th>release</th>\n",
       "      <th>artist_name</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>d6589314c0a9bcbca4fee0c93b14bc402363afea</td>\n",
       "      <td>SOADQPP12A67020C82</td>\n",
       "      <td>12</td>\n",
       "      <td>You And Me Jesus</td>\n",
       "      <td>Tribute To Jake Hess</td>\n",
       "      <td>Jake Hess</td>\n",
       "      <td>2004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d6589314c0a9bcbca4fee0c93b14bc402363afea</td>\n",
       "      <td>SOAFTRR12AF72A8D4D</td>\n",
       "      <td>1</td>\n",
       "      <td>Harder Better Faster Stronger</td>\n",
       "      <td>Discovery</td>\n",
       "      <td>Daft Punk</td>\n",
       "      <td>2007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>d6589314c0a9bcbca4fee0c93b14bc402363afea</td>\n",
       "      <td>SOANQFY12AB0183239</td>\n",
       "      <td>1</td>\n",
       "      <td>Uprising</td>\n",
       "      <td>Uprising</td>\n",
       "      <td>Muse</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>d6589314c0a9bcbca4fee0c93b14bc402363afea</td>\n",
       "      <td>SOAYATB12A6701FD50</td>\n",
       "      <td>1</td>\n",
       "      <td>Breakfast At Tiffany's</td>\n",
       "      <td>Home</td>\n",
       "      <td>Deep Blue Something</td>\n",
       "      <td>1993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>d6589314c0a9bcbca4fee0c93b14bc402363afea</td>\n",
       "      <td>SOBOAFP12A8C131F36</td>\n",
       "      <td>7</td>\n",
       "      <td>Lucky (Album Version)</td>\n",
       "      <td>We Sing.  We Dance.  We Steal Things.</td>\n",
       "      <td>Jason Mraz &amp; Colbie Caillat</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       user                song  listen_count  \\\n",
       "0  d6589314c0a9bcbca4fee0c93b14bc402363afea  SOADQPP12A67020C82            12   \n",
       "1  d6589314c0a9bcbca4fee0c93b14bc402363afea  SOAFTRR12AF72A8D4D             1   \n",
       "2  d6589314c0a9bcbca4fee0c93b14bc402363afea  SOANQFY12AB0183239             1   \n",
       "3  d6589314c0a9bcbca4fee0c93b14bc402363afea  SOAYATB12A6701FD50             1   \n",
       "4  d6589314c0a9bcbca4fee0c93b14bc402363afea  SOBOAFP12A8C131F36             7   \n",
       "\n",
       "                           title                                release  \\\n",
       "0               You And Me Jesus                   Tribute To Jake Hess   \n",
       "1  Harder Better Faster Stronger                              Discovery   \n",
       "2                       Uprising                               Uprising   \n",
       "3         Breakfast At Tiffany's                                   Home   \n",
       "4          Lucky (Album Version)  We Sing.  We Dance.  We Steal Things.   \n",
       "\n",
       "                   artist_name  year  \n",
       "0                    Jake Hess  2004  \n",
       "1                    Daft Punk  2007  \n",
       "2                         Muse     0  \n",
       "3          Deep Blue Something  1993  \n",
       "4  Jason Mraz & Colbie Caillat     0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 数据集太大，训练时间过长，选取前10000条数据\n",
    "data = pd.read_csv('./new_data/triplet_dataset_sub_song_merged.csv', nrows = 10000)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 7)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>listen_count</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.886400</td>\n",
       "      <td>1670.893700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>20.219901</td>\n",
       "      <td>743.648891</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1990.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>2003.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>4.000000</td>\n",
       "      <td>2007.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1890.000000</td>\n",
       "      <td>2010.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       listen_count          year\n",
       "count  10000.000000  10000.000000\n",
       "mean       3.886400   1670.893700\n",
       "std       20.219901    743.648891\n",
       "min        1.000000      0.000000\n",
       "25%        1.000000   1990.000000\n",
       "50%        1.000000   2003.000000\n",
       "75%        4.000000   2007.000000\n",
       "max     1890.000000   2010.000000"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.to_csv('data_small.csv', index = False)\n",
    "data.describe()   # 观察数据的情况"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 分割训练数据和测试数据\n",
    "from sklearn.model_selection import train_test_split\n",
    "data_train, data_test = train_test_split(data, random_state = 21, test_size = 0.2)\n",
    "data_train.to_csv('data_small_train.csv', index = False)\n",
    "data_test.to_csv('data_small_test.csv', index = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 基于用户的协同过滤"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 定义一些变量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 1. 从全部数据集获得不重复的用户和不重复的歌曲数量\n",
    "uniqueUsers = list(data_train['user'].unique())\n",
    "uniqueSounds = list(data_train['song'].unique())\n",
    "\n",
    "user_num = len(uniqueUsers)\n",
    "sound_num = len(uniqueSounds)\n",
    "\n",
    "\n",
    "# 2. 每个用户听过的歌曲，听过每首歌曲的用户 的对照字典\n",
    "soundsForUser = defaultdict(set)  \n",
    "usersForSound = defaultdict(set)  \n",
    "\n",
    "# 对训练集的用户进行统计\n",
    "f_data_train = open('data_small_train.csv', 'r')\n",
    "f_data_train.readline() #跳过第一行，标题行\n",
    "for line in f_data_train:\n",
    "    cols = line.strip().split(\",\")  #将数据按，分开\n",
    "    # 该用户参加了这个活动，该活动的参加用户\n",
    "    soundsForUser[cols[0]].add(cols[1])   \n",
    "    usersForSound[cols[1]].add(cols[0])\n",
    "f_data_train.close()\n",
    "\n",
    "\n",
    "# 3. 编码用户索引和歌曲索引字典\n",
    "userIndex = dict()\n",
    "soundIndex = dict()\n",
    "        \n",
    "#重新编码用户索引字典\n",
    "for i, u in enumerate(uniqueUsers):\n",
    "    userIndex[u] = i\n",
    "            \n",
    "#重新编码歌曲索引字典\n",
    "for i, s in enumerate(uniqueSounds):\n",
    "    soundIndex[s] = i\n",
    "        \n",
    "        \n",
    "# 4. 歌曲打分矩阵（用户——歌曲：听过 = 1， 没听过 = 0）\n",
    "userSoundScores = np.matlib.zeros((len(uniqueUsers), len(uniqueSounds)))\n",
    "# 听歌次数矩阵（用户——歌曲：次数）\n",
    "userSoundTimes = np.matlib.zeros((len(uniqueUsers), len(uniqueSounds)))\n",
    "\n",
    "ftrain = open(\"data_small_train.csv\", 'r')\n",
    "ftrain.readline()  # 跳过标题行\n",
    "for line in ftrain:\n",
    "    cols = line.strip().split(\",\")   # 每行数据按逗号分开\n",
    "    i = userIndex[cols[0]]   #用户编号\n",
    "    j = soundIndex[cols[1]]  #歌曲编号\n",
    "    \n",
    "    if cols[2] is not 0:\n",
    "        userSoundScores[i, j] = 1\n",
    "        userSoundTimes[i, j] = cols[2]\n",
    "    else:\n",
    "        userSoundScores[i, j] = 0\n",
    "ftrain.close()                                             "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    " # 5. 将用户组成两两一组，同时将歌曲两两一组，之后相似度矩阵用\n",
    "uniqueUserPairs = set()\n",
    "uniqueSoundPairs = set()\n",
    "for sound in usersForSound.keys():\n",
    "    users = usersForSound[sound]\n",
    "    if len(users) > 2:\n",
    "        uniqueUserPairs.update(itertools.combinations(users, 2))\n",
    "        \n",
    "for user in soundsForUser.keys():\n",
    "    sounds = soundsForUser[user]\n",
    "    if len(sounds) > 2:\n",
    "        uniqueSoundPairs.update(itertools.combinations(sounds, 2))\n",
    "# uniqueSounds的长度和usersForSound.keys()不同，上面用户组循环中若使用uniqueSounds，会改变usersForSound.keys()长度\n",
    "# print(len(uniqueSounds), len(usersForSound.keys()))  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of users: 84\n",
      "number of sounds: 5862\n",
      "user pairs:  1186\n",
      "sound pairs:  569152\n"
     ]
    }
   ],
   "source": [
    "# 查看生成的数据\n",
    "print ('number of users:', len(uniqueUsers))\n",
    "print ('number of sounds:', len(uniqueSounds))\n",
    "print ('user pairs: ', len(uniqueUserPairs))\n",
    "print ('sound pairs: ', len(uniqueSoundPairs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 计算用户的相似度\n",
    "相似度：两个用户播放歌曲的交集/两个用户播放歌曲的并集（将歌曲的播放次数定义为0/1时）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 建立用户相似度矩阵，将矩阵的数值填充为nan，方便后面计算推荐矩阵的中值\n",
    "userSimMatrix = np.matlib.zeros((user_num, user_num))\n",
    "userSimMatrix[:] = np.nan \n",
    "            \n",
    "#对称\n",
    "for u1, u2 in uniqueUserPairs:\n",
    "    i = userIndex[u1]   # u1的index\n",
    "    j = userIndex[u2]   # 从定义的 index dict 找到u2的index\n",
    "    if np.isnan(userSimMatrix[i,j]):      # 如果相似度矩阵的两两用户对应位置还没有相似度数值\n",
    "        sim_user = len(soundsForUser[u1]&soundsForUser[u2]) / len(soundsForUser[u1]|soundsForUser[u2])\n",
    "        userSimMatrix[i, j] = sim_user      # 在用户相似度矩阵对应的位置赋值相似度\n",
    "        userSimMatrix[j, i] = sim_user      # 对称位置也赋值\n",
    "        \n",
    "# sio.mmwrite(\"./new_data/userSimMatrix\", userSimMatrix)    # 保存相似度矩阵文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 定义基于用户的协同过滤函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 定义基于用户的协同过滤函数\n",
    "def userCFReco(userId, soundId):\n",
    "    sim_accumulate = 0.0  \n",
    "    rat_acc = 0.0  \n",
    "\n",
    "    for user in usersForSound[soundId]:  # 听过参数歌曲的所有用户\n",
    "        # 参数user与歌曲的打分用户之间的相似度\n",
    "        sim = len(soundsForUser[user]&soundsForUser[userId]) / len(soundsForUser[user]|soundsForUser[userId])    \n",
    "            \n",
    "        # 预测参数user对歌曲的打分 = 参数user与给歌曲打过分的关联用户的相似度 * 关联用户的打分\n",
    "        if sim != 0:\n",
    "            rat_acc += sim * userSoundScores[userIndex[user], soundIndex[soundId]]   \n",
    "\n",
    "       # sim_accumulate += sim  # 累加参数用户 与 给参数歌曲打过分用户中的关联用户 的相似度\n",
    "     \n",
    "    if len(usersForSound[soundId]) == 0:\n",
    "        ans = 0\n",
    "    else:\n",
    "        ans = rat_acc / len(usersForSound[soundId])     # 预测结果为：打分预测 / 关联用户数量\n",
    "\n",
    "        \n",
    "    # 为了之后计算推荐的误差和准确率，返回预测推荐度的值\n",
    "    return ans  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 基于歌曲的协同过滤"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 计算歌曲之间的相似度\n",
    "_相似度：播放两个歌曲的用户交集 / 播放两个歌曲的用户并集_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 计算歌曲的相似度\n",
    "num_sound = len(uniqueSounds)\n",
    "soundSimMatrix = np.matlib.zeros((num_sound, num_sound))\n",
    "soundSimMatrix[:] = np.nan\n",
    "            \n",
    "#对称\n",
    "for s1, s2 in uniqueSoundPairs:\n",
    "    i = soundIndex[s1]   # 歌曲1的index\n",
    "    j = soundIndex[s2]   # 歌曲2的index\n",
    "    if np.isnan(soundSimMatrix[i,j]):      # 如果相似度不为0\n",
    "        # 相似度：播放两个歌曲的用户交集 / 播放两个歌曲的用户并集\n",
    "        sim_sound = len(usersForSound[s1]&usersForSound[s2]) / len(usersForSound[s1]|usersForSound[s2])\n",
    "        soundSimMatrix[i, j] = sim_sound      # 在歌曲相似度矩阵对应的位置赋值\n",
    "        soundSimMatrix[j, i] = sim_sound      # 对称位置也赋值\n",
    "        \n",
    "# sio.mmwrite(\"./new_data/soundSimMatrix\", soundSimMatrix)    # 保存相似度矩阵文件，以备之后使用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 定义基于歌曲协同过滤的函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 根据item-based协同过滤，得到sound的推荐度\n",
    "def soundCFReco(userId, soundId):\n",
    "    sim_accumulate = 0.0  \n",
    "    rat_acc = 0.0  \n",
    "\n",
    "    for sound in soundsForUser[userId]:  # 参数用户听过的所有歌曲\n",
    "        # 相似度：参数歌曲的用户与参数用户听过的歌曲的其他用户的交集/并集\n",
    "        sim = len(usersForSound[sound]&usersForSound[soundId]) / len(usersForSound[sound]|usersForSound[soundId])     \n",
    "            \n",
    "        # 参数歌曲的预测打分 = 参数user对其他歌曲的打分 * 关联歌曲的相似度\n",
    "        if sim != 0:\n",
    "            rat_acc += sim * userSoundScores[userIndex[userId], soundIndex[sound]]   \n",
    "\n",
    "       # sim_accumulate += sim  # 累加相似度\n",
    "     \n",
    "    if len(soundsForUser[userId]) == 0:\n",
    "        ans = 0\n",
    "    else:\n",
    "        ans = rat_acc / len(soundsForUser[userId])     # 预测结果为：打分预测 / 关联歌曲数量\n",
    "\n",
    "    return ans   # 直接范围预测值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 基于模型的协同过滤"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 定义基于SVD模型协同过滤的函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def svdCFReco(dataMatrix, simMeas, percentage):\n",
    "    # 预测的推荐歌曲的结果矩阵\n",
    "    score_SVD_matrix = userSoundScores.copy()\n",
    "    [nuser, n] = shape(dataMatrix)    # 用户数和歌曲数量 （从用户听歌数量矩阵获得）\n",
    "    \n",
    "    # SVD 分解矩阵：用线性代数包计算\n",
    "    u, sigma, vt = la.svd(dataMatrix)\n",
    "    k = sigmaPct(sigma, percentage) # k：通过中间的奇异值矩阵的大小确定\n",
    "    sigmaK = mat(eye(k)*sigma[:k])  # 转换为对角矩阵（当前的sigma是向量）\n",
    "    #根据k的值将原始数据转换到k维空间(低维),xformedItems表示物品(item)在k维空间转换后的值\n",
    "    xformedItems = dataMatrix.T * u[:,:k] * sigmaK.I  \n",
    "    \n",
    "    # 计算是否推荐歌曲，并将结果存入‘预测的推荐歌曲的结果矩阵’\n",
    "    for user in range(nuser):\n",
    "        unlistened = nonzero(userSoundScores[user,:].A==0)[1]   # 当前用户没有听过的歌曲list\n",
    "        score_SVD_list = list()     # 基于模型计算没听过的歌曲的打分list\n",
    "        for sound_index in unlistened:   # 每首歌曲index\n",
    "            simTotal = 0.0     # 相似度\n",
    "            ratSimTotal = 0.0  # 基于相似度的预测打分\n",
    "            for j in range(n):   # 对于每一首歌j\n",
    "                userRating = dataMatrix[user,j]   # 从用户——歌曲次数矩阵获得听歌次数\n",
    "                if userRating == 0 or j == sound_index : continue  # 如果是没听过的歌，或者是当前歌曲自身，继续下一首\n",
    "                # 计算当前歌曲与之前听过歌曲j之间的相似度：cos夹角\n",
    "                similarity = simMeas(xformedItems[sound_index,:].T, xformedItems[j,:].T)\n",
    "                simTotal += similarity   # 对与所有歌曲相似度求和\n",
    "                ratSimTotal += similarity*userRating  # 当前歌曲和听过歌曲j的相似度 * 用户听过歌曲j的次数，并求和\n",
    "            if simTotal == 0:\n",
    "                score_SVD = 0\n",
    "            else:\n",
    "                score_SVD = ratSimTotal / simTotal # 预测当前歌曲被听次数/分数\n",
    "            score_SVD_list.append((sound_index, score_SVD))   # 将预测打分存入list\n",
    "\n",
    "        # 当前用户对所有歌曲打分的排序\n",
    "        score_SVD_list = sorted(score_SVD_list, key = lambda x : x[1], reverse = True)\n",
    "\n",
    "        # top？的歌曲打分存入用户——歌曲打分matrix（0/1）\n",
    "        for score in score_userCFReco_list[:numberRecSound]:\n",
    "            score_SVD_matrix[user, score[0]] = 1\n",
    "    return score_SVD_matrix\n",
    "    \n",
    "\n",
    "# 计算歌曲相似度函数：cos夹角\n",
    "def cosSim(inA, inB):\n",
    "    num = float(inA.T * inB)  # 两个向量的内积\n",
    "    denom = la.norm(inA) * la.norm(inB)  # 两向量的模的积\n",
    "    return 0.5+0.5*(num/denom)  # 将相似度归一到0与1之间\n",
    "\n",
    "# 定义求sigma矩阵k的值的函数\n",
    "def sigmaPct(sigma, percentage):\n",
    "    sigma2 = sigma**2     # 对sigma求平方：sigma作为向量代入\n",
    "    sumsgm2 = sum(sigma2) # 求所有奇异值sigma的平方和\n",
    "    sumsgm3 = 0      # sumsgm3是前k个奇异值的平方和\n",
    "    \n",
    "    k = 0   \n",
    "    for i in sigma:\n",
    "        sumsgm3 += i**2\n",
    "        k += 1\n",
    "        if sumsgm3 >= sumsgm2 * percentage:    # 当k个sigma > 目标比例时，返回k（相当于降维）\n",
    "            return k"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 在测试集上进行预测并评价\n",
    "_准确率：推荐的歌曲中用户感兴趣的比例_  \n",
    "_召回率：用户感兴趣的歌曲中被推荐到的比例_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 定义评价函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 计算评价时要用的top歌曲数量\n",
    "numberRecSound = 50\n",
    "# 对测试集的用户进行预测\n",
    "uniqueUsers_test = list(data_test['user'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 从推荐结果里选top？歌曲的准确率和召回率进行评价\n",
    "# 建立测试用户和预测打分的矩阵\n",
    "score_userCFReco_matrix = userSoundScores.copy()\n",
    "score_soundCFReco_matrix = userSoundScores.copy()\n",
    "\n",
    "for user in uniqueUsers:   # 每个用户\n",
    "    # 获得当前用户没有打分的歌曲的index\n",
    "    unlistened = nonzero(userSoundScores[userIndex[user],:].A==0)[1]\n",
    "    # 建立打分list存放每个用户对每首歌的预测打分结果\n",
    "    score_userCFReco_list = list()\n",
    "    score_soundCFReco_list = list()\n",
    "    for sound_index in unlistened:   # 每首歌曲\n",
    "        score_userCFReco = userCFReco(user, list(soundIndex.keys())[list(soundIndex.values()).index(sound_index)])\n",
    "        score_userCFReco_list.append((sound_index, score_userCFReco))   # 将预测打分存入list\n",
    "        \n",
    "        score_soundCFReco = soundCFReco(user, list(soundIndex.keys())[list(soundIndex.values()).index(sound_index)])\n",
    "        score_soundCFReco_list.append((sound_index, score_soundCFReco))   # 将预测打分存入list\n",
    "\n",
    "    # 当前用户对所有歌曲打分的排序\n",
    "    score_userCFReco_list = sorted(score_userCFReco_list, key = lambda x : x[1], reverse = True)\n",
    "    score_soundCFReco_list = sorted(score_soundCFReco_list, key = lambda x : x[1], reverse = True)\n",
    "    \n",
    "    # top10的歌曲打分存入matrix\n",
    "    for score in score_userCFReco_list[:numberRecSound]:\n",
    "        score_userCFReco_matrix[userIndex[user], score[0]] = 1\n",
    "    \n",
    "    # top10的歌曲打分存入matrix\n",
    "    for score in score_soundCFReco_list[:numberRecSound]:\n",
    "        score_soundCFReco_matrix[userIndex[user], score[0]] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 评价：计算准确率和召回率（测试集和top50的歌曲）\n",
    "def measurement(matrix):\n",
    "    accurate_pred_num = 0   # 用户感兴趣的歌曲中被推荐的数量\n",
    "\n",
    "    f = open('data_small_test.csv', 'r')\n",
    "    f.readline() #跳过第一行，标题行\n",
    "\n",
    "    for line in f:\n",
    "        cols = line.strip().split(\",\")  #将数据按，分开\n",
    "        if cols[0] in userIndex and cols[1] in soundIndex:  # 如果不是新用户/新歌曲，新用户/新歌不考虑\n",
    "            # 如果该歌曲被推荐，则 accurate_pred_num + 1\n",
    "            if matrix[userIndex [cols[0]], soundIndex[cols[1]]] == 1:\n",
    "                accurate_pred_num += 1\n",
    "    f.close() \n",
    "    \n",
    "    # 准确率 = 所有用户感兴趣并且被推荐的歌曲数 / 推荐的歌曲总数 \n",
    "    precision = float(accurate_pred_num / (len(uniqueUsers_test) * numberRecSound))\n",
    "    \n",
    "    # 召回率 = 所有用户感兴趣并且被推荐的歌曲数 / 所有用户感兴趣的歌曲总数\n",
    "    recall = float(accurate_pred_num / data_test.shape[0])\n",
    "    \n",
    "    return precision, recall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 调用函数评价"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((0.014216867469879518, 0.0295), (0.02674698795180723, 0.0555))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 基于用户协同过滤的准确率和召回率，基于歌曲的协同过滤的准确率和召回率\n",
    "print('基于用户，歌曲协同过滤的准确率和召回率：', measurement(score_userCFReco_matrix), measurement(score_soundCFReco_matrix))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 调用SVD模型函数预测，sigma矩阵k值 = 0.9\n",
    "score_SVD_matrix = svdCFReco(userSoundTimes, cosSim, 0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0007228915662650603, 0.0015)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 调用评价函数计算准确率和召回率\n",
    "print('基于svd模型协同过滤的准确率和召回率：', measurement(score_SVD_matrix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "准确率和召回率结果比较：   \n",
    "user_CF  &emsp; 0.014216867469879518, &emsp; 0.0295    \n",
    "item_CF  &emsp; 0.02674698795180723,  &ensp;&emsp; 0.0555       \n",
    " svd_CF  &ensp;&emsp; 0.0007228915662650603, &ensp; 0.0015"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "top歌曲评价结果：   \n",
    "1. top50歌曲的话，基于歌曲的协同过滤的准确率要高于基于用户的协同过滤      \n",
    "2. 推荐top10歌曲的话，准确率为0。可能因为数据集太小，增加数据到20000时有改善\n",
    "3. 模型的评价结果不是很好，可能是10000数据比较小，模型参数使用缺省参数的原因？\n",
    "4. 有的用户在训练集数据里听过的歌很少，在用测试集推荐时，可能好多歌曲的预测打分是一样的，但只用top10/top50做评价，还有好多歌曲的也是top10/50，但是会被忽略。所以用更多的推荐歌曲做评价会更准确？"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
